Systems and methods for streaming XPath query

ABSTRACT

An improved XML query system represents an XML document as a stream of discrete ‘events,’ with each event representing a portion of the document as the document is being parsed. Expression-based event matching such as XPath can be performed against the event stream using a stack to keep only the relevant contexts in memory. Observers can be used to listen for matching events. Matching events can then be routed for processing by appropriate objects or components and returned to the event stream if necessary.  
     This description is not intended to be a complete description of, or limit the scope of, the invention. Other features, aspects, and objects of the invention can be obtained from a review of the specification, the figures, and the claims.

CLAIM OF PRIORITY

[0001] This application claims priority from the following application,which is hereby incorporated by reference in its entirety:

[0002] U.S. Provisional Application No. 60/451,313, entitled SYSTEMS ANDMETHODS FOR STREAMING XPATH QUERY, by Patrick Calahan, filed on Feb. 28,2003 (Attorney Docket No. BEAS-01330US0 SRM/DTX).

CROSS-REFERENCED CASES

[0003] The following applications are cross-referenced and incorporatedherein by reference in its entirety:

[0004] U.S. patent application Ser. No. 10/304,207 entitled “StreamingParser API,” by Chris Fry et al., filed Nov. 26, 2002.

COPYRIGHT NOTICE

[0005] A portion of the disclosure of this patent document containsmaterial which is subject to copyright protection. The copyright ownerhas no objection to the facsimile reproduction by anyone of the patentdocument of the patent disclosure, as it appears in the Patent andTrademark Office patent file or records, but otherwise reserves allcopyright rights whatsoever.

FIELD OF THE INVENTION

[0006] The present invention relates to the querying of data, such asfrom a document or file.

BACKGROUND

[0007] XPath is a W3C language standard that can be used to address orquery parts of an XML document. It models an XML document as a tree ofnodes, which can include element nodes, attribute nodes and/or textnodes. XPath can be used to identify a subset of an XML document bymatching, or determining whether a node matches a pattern, similar tohow SQL can be used against a database. In the typical case, anexpression written in the XPath language is evaluated against an XMLdocument to determine which parts of the document ‘match’ the XPath. Inorder to do this, the XML document must be parsed and represented inmemory. One of the standard representations of XML is the DocumentObject Model (DOM). DOM model presents an XML document as a hierarchy ofnodes through which one can navigate arbitrarily. This approach providesa lot of flexibility, but comes at a cost in terms of efficiency andmemory use, as the entire document must be brought into memory at onetime.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008]FIG. 1 is a diagram showing an exemplary system that can be usedin accordance with one embodiment of the present invention.

[0009]FIG. 2 shows an exemplary data tree that can be used with thesystem of FIG. 1 in an embodiment.

[0010]FIG. 3 is a flowchart for an exemplary process that can be usedwith the system of FIG. 1 in an embodiment.

DETAILED DESCRIPTION

[0011] The invention is illustrated by way of example and not by way oflimitation in the figures of the accompanying drawings in which likereferences indicate similar elements. It should be noted that referencesto “an” or “one” embodiment in this disclosure are not necessarily tothe same embodiment, and such references mean at least one.

[0012] Systems and methods in accordance with one embodiment of thepresent invention overcome deficiencies in existing XML query systems byrepresenting the XML document as a stream of discrete ‘events’, witheach event representing a portion of the document as the document isbeing parsed. Event matching can be performed against the event stream.Matching events can then be routed for processing by appropriate objectsor components and returned to the event stream if necessary.

[0013] XPath can be used to identify a subset of an XML document,similar to how SQL can be used against a database. XPath is a W3Clanguage standard that can be used to address or query parts of an XMLdocument. It can address parts of an XML document by providing basicfacilities for manipulating strings, numbers, and Boolean variables.XPath operates on the hierarchical structure, which can be but is notlimited to a tree, instead of the syntax of an XML document and can beused for matching, or determining whether a node matches a pattern. Itmodels an XML document as a tree of nodes, which can include elementnodes, attribute nodes and/or text nodes and defines a way to compute astring-value for each node type. The primary syntactic construct inXPath is the expression. An expression is evaluated to yield an objectof type node-set, Boolean, number, or string. In the typical case, anexpression written in the XPath language is evaluated against an XMLdocument to determine which parts of the document ‘match’ the XPath. Inorder to do this, the XML document must be parsed and represented inmemory.

[0014] Systems and methods in accordance with one embodiment of thepresent invention adopt a true streaming approach, passing bits of anXML document one after another, and it is up to the system to decidewhat to do with each bit as it passes on the stream. An advantage of atrue streaming approach is that such a system is faster and far morememory efficient than a DOM-style approach, since only one portion ofthe document is in memory at any given time. When using a streamingparser, a system can take a stream on an XML document, generating astream of events, one event for each node in the XML tree, and performXPath matching on that stream. A streaming XPath system can also beschema aware, such that the system knows the XML schema for a document,that schema can be used to provide insight on how to most effectivelyprocess the document. For instance, the need to go “backwards” in astream can be avoided if the system knows in advance which events itneeds to grab and in what order those events will be received.

[0015] A streaming approach can place a greater burden on a system tomaintain relevant state than a DOM approach, as a streaming approach mayprovide no navigation mechanisms. While such an approach provides a veryefficient way to process an XML document, the efficiency comes at acost, as there can be considerably less context available when workingwith a stream than when working with a DOM tree. Further, XPath has tobe able to traverse the hierarchy, in some sense, in order to locate theappropriate portion of the document. In many instances, it is simple tolocate an appropriate portion of XML against a DOM tree, since thesystem is able to walk against the tree. When using a stream, a systemhas to maintain context in a way that is efficient enough to make usingthe stream worthwhile. Some tradeoffs can be made, such as notsupporting the entire XPath specification. At some point, it may be moreefficient to realize an entire DOM tree, if doing a convoluted matchingagainst the entire document.

[0016] The XPath specification defines the notion of a context, where acontext is the information about an event, consisting of a node itrepresents, a position of the node relative to a parent node, and afunction library, as well as any of several other components such asvariable bindings. A location path is a type of expression that canselect a set of nodes relative to the context node. The evaluation of alocation path expression can result in the node-set containing the nodesbeing selected by the location path. Location paths can recursivelycontain expressions used to filter node sets. Expressions can be parsedby first dividing the character string to be parsed into tokens, thenparsing the resulting token sequence.

[0017] In one embodiment, it is relatively easy to map context to thestream, as the system can maintain a stack of stream events that providethe direct ancestral line back to the root. For instance, matching anXPath that consists solely of child axes can be straightforward. Inanother embodiment, mapping can become more complicated in the case ofdescendant axes, similar to matching an entire sub-tree. In those cases,it can be necessary to spawn a tree of contexts and perform matchingagainst each of those contexts. It can become complicated, as the systemgets to maintain, and know when you can discard those cloned contexts.It can be even more complicated when matching axes called “following,”which match everything below a certain point in the document. In somecases, it is necessary to maintain that context tree and track what toadd on to the tree as the system navigates its way back out of thedocument.

[0018] Systems and methods in accordance with one embodiment of thepresent invention know how to manage the multi-context mode discussed inthe proceeding paragraph. They utilize the information of contexts inthe stack matching against the expression to recognize when to go intothis multi-context mode, when to destroy those contexts, and how toupdate the context stack appropriately. Certain optimizations can alsobe used that can know when not to match certain contexts in the contexttree. XPath defines different ways to slice up a document, such asparents and children, that each has to be dealt with in a different way.

[0019] Systems and methods in accordance with one embodiment do notaccount for reverse axes. A reverse axis is any axis that would requiregoing “back” through the stream. A diagram showing an exemplary“forward” and “backward” or “reverse” path through a data tree is givenby FIG. 2. A diagram of an exemplary system is shown in FIG. 1. Astreaming parser 102 generates events by parsing an XML document 100,and then places those events on an XML event stream. Such a streamingprocess is demonstrated by the diagram of FIG. 3. The streaming parserfirst takes a tree of an XML document as the input 300, traverses theXML tree either through a broad-first search or a depth-first search andadds each node visited into a data structure, e.g., a queue 302. Thestreaming parser then processes the queue in the first-in-first-out(FIFO) manner 304 to generate an event for the context of each node inthe queue 306 and appends each event to the output stream 308. Using theevent stream, the end user of the streaming API pulls events from thestream as they come through it. When a user calls for the next event onthe stream, that user has a guarantee that they will get the next event.The user will find out if the next event is going to match, and willfind out before the call to next returns.

[0020] In one embodiment, an XPath matching component 104 performsmatching on each event received on the stream. Matching can becommunicated to a caller or end user in a number of ways. These systemsare doing event-based processing, as opposed to static tree-basedprocessing. In a tree-based implementation, for example, a user canrequest all the nodes that match an XPath for a document. The user willreceive a collection of nodes that match that XPath. Such an approach isnot necessarily effective in the case of streaming, as it is thennecessary to read through the document, save all the nodes, and presentthe collection to the user. This is fundamentally not a stream-centricway of looking at the problem. Instead, using an XPath matchingapproach, an observer 106 can be registered. The registered observer isan object to be notified whenever an event comes through the stream thatmatches this Xpath. If an event matches an XPath, that event can betemporarily diverted and sent over to a user-defined object 108 thatreacts to the match. Then, the event can be returned to the stream ifnecessary so that any subsequent object pulling events from the streamcan process that event.

[0021] One embodiment may be implemented using a conventional generalpurpose or a specialized digital computer or microprocessor(s)programmed according to the teachings of the present disclosure, as willbe apparent to those skilled in the computer art. Appropriate softwarecoding can readily be prepared by skilled programmers based on theteachings of the present disclosure, as will be apparent to thoseskilled in the software art. The invention may also be implemented bythe preparation of integrated circuits or by interconnecting anappropriate network of conventional component circuits, as will bereadily apparent to those skilled in the art.

[0022] One embodiment includes a computer program product which is astorage medium (media) having instructions stored thereon/in which canbe used to program a computer to perform any of the features presentedherein. The storage medium can include, but is not limited to, any typeof disk including floppy disks, optical discs, DVD, CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs,VRAMs, flash memory devices, magnetic or optical cards, nanosystems(including molecular memory ICs), or any type of media or devicesuitable for storing instructions and/or data.

[0023] Stored on any one of the computer readable medium (media), thepresent invention includes software for controlling both the hardware ofthe general purpose/specialized computer or microprocessor, and forenabling the computer or microprocessor to interact with a human user orother mechanism utilizing the results of the present invention. Suchsoftware may include, but is not limited to, device drivers, operatingsystems, execution environments/containers, and applications.

[0024] The foregoing description of the preferred embodiments of thepresent invention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Many modifications andvariations will be apparent to the practitioner skilled in the art.Embodiments were chosen and described in order to best describe theprinciples of the invention and its practical application, therebyenabling others skilled in the art to understand the invention, thevarious embodiments and with various modifications that are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. A system to process an XML document, comprising:a streaming parser capable of parsing an XML document and generating astream of at least one event, wherein each event can represent a portionof the document; a matching component capable of performing matching onan event in the stream and notifying an observer if the event is amatch; said observer capable of listening for a matching event andpassing it to a user object; and said user object capable of handlingthe matching event.
 2. The system according to claim 1, wherein: the XMLdocument is represented in a hierarchical structure.
 3. The systemaccording to claim 2, wherein: the hierarchical structure can be a treewith each node containing a portion of the document.
 4. The systemaccording to claim 3, wherein: the streaming parser is capable ofperforming a method, comprising: traversing the XML tree and addingvisited nodes into a data structure; processing the nodes in the datastructure and generating an event for each node; and appending the eventto the output stream.
 5. The system according to claim 4, wherein: thetree can be traversed using a breath-first or depth-first search.
 6. Thesystem according to claim 4, wherein: the data structure can be a queue.7. The system according to claim 4, wherein: the data structure can beprocessed using a first-in-first-out approach.
 8. The system accordingto claim 1, wherein: the matching component is capable of keeping only aportion of the XML document in memory at any given time.
 9. The systemaccording to claim 1, wherein: the matching component is capable ofknowing the schema of the XML document and foreseeing the coming events.10. The system according to claim 1, wherein: the matching component iscapable of performing an expression-based match, which can be an XPathquery.
 11. The system according to claim 3, wherein: the matchingcomponent is capable of keeping, cloning and destroying the entirety ora portion of the sub-tree descending from a node in the tree.
 12. Thesystem according to claim 1, wherein: the user object is capable ofreturning the matching event to an XML stream for use by any othercomponent.
 13. A method for processing an XML document, comprising:parsing an XML document and generating a stream of at least one event,wherein each event can represent a portion of the document; performingmatching on an event in the stream and notifying an observer if theevent is a match; listening for a matching event and passing it to auser object; and handling the matching event.
 14. The method accordingto claim 13, further comprising: representing the XML document in ahierarchical structure, which can be a tree with each node containing aportion of the document.
 15. The method according to claim 14, wherein:the parsing of the XML document comprises the steps of: traversing theXML tree and adding visited nodes into a data structure; processing thenodes in the data structure and generating an event for each node; andappending the event to the output stream.
 16. The method according toclaim 15, wherein: the XML tree is traversed using a breath-first ordepth-first search.
 17. The method according to claim 15, wherein: thedata structure is processed using a first-in-first-out approach.
 18. Themethod according to claim 13, further comprising: keeping only a portionof the XML document in memory at any given time.
 19. The methodaccording to claim 13, further comprising: knowing the schema of the XMLdocument and foreseeing the coming events.
 20. The method according toclaim 13, further comprising: performing an expression-based match,which can be an XPath query.
 21. The method according to claim 14,further comprising: keeping, cloning and destroying the entirety or aportion of the sub-tree descending from a node in the tree.
 22. Themethod according to claim 13, further comprising: returning the matchingevent to an XML stream for use by any other component.
 23. A machinereadable medium having instructions stored thereon that when executed bya processor cause a system to: parse an XML document and generate astream of at least one event, wherein each event can represent a portionof the document; perform matching on an event in the stream and notifyan observer if the event is a match; listen for a matching event andpass it to a user object; and handle the matching event.
 24. The machinereadable medium of claim 23, further comprising instructions that whenexecuted cause the system to: represent the XML document in ahierarchical structure, which can be a tree with each node containing aportion of the document.
 25. The machine readable medium of claim 24,wherein the instructions that when executed cause the system to: parsethe XML document, comprising the steps of: traversing the XML tree andadding visited nodes into a data structure; processing the nodes in thedata structure and generating an event for each node; and appending theevent to the output stream.
 26. The machine readable medium of claim 25,wherein the instructions that when executed cause the system to:traverse the tree using a breath-first or depth-first search.
 27. Themachine readable medium of claim 25, wherein the instructions that whenexecuted cause the system to: process the data structure using afirst-in-first-out approach.
 28. The machine readable medium of claim23, further comprising instructions that when executed cause the systemto: perform an expression-based match, which can be an XPath query. 29.The machine readable medium of claim 23, further comprising instructionsthat when executed cause the system to: keep only a portion of the XMLdocument in memory at any given time.
 30. The machine readable medium ofclaim 23, further comprising instructions that when executed cause thesystem to: know the schema of the XML document and foresee the comingevents.
 31. The machine readable medium of claim 24, further comprisinginstructions that when executed cause the system to: keep, clone anddestroy the entirety or a portion of the sub-tree descending from a nodein the tree.
 32. The machine readable medium of claim 23, furthercomprising instructions that when executed cause the system to: returnthe matching event to an XML stream for use by any other component. 33.A system for processing an XML document, comprising: means for parsingan XML document and generating a stream of at least one event, whereineach event can represent a portion of the document; means for performingmatching on an event in the stream and notifying an observer if theevent is a match; means for listening for a matching event and passingit to a user object; and means for handling the matching event.
 34. Acomputer data signal embodied in a transmission medium, comprising: acode segment including instructions to parse an XML document andgenerate a stream of at least one event, wherein each event canrepresent a portion of the document; a code segment includinginstructions to perform matching on an event in the stream and notify anobserver if the event is a match; a code segment including instructionsto listen for a matching event and pass it to a user object; and a codesegment including instructions to handle the matching event.